® Check for updates 


AERA Open 

July-September 2019, Vol. 5, No. 3, pp. 1-21 

DOI: 10.1177/2332858419867290 

Article reuse guidelines: sagepub.com/journals-permissions 
© The Author(s) 2019. http:/journals.sagepub.com/home/ero 


Long-Term Effects of Social—Emotional Learning on Receipt of Special 
Education and Grade Retention: Evidence From a Randomized Trial of 
INSIGHTS 


Meghan P. McCormick 


MDRC 
Robin Neuhaus 

E. Parham Horn 
Erin E. O’Connor 
New York University 
Hope S. White 

SUNY Buffalo 

Samantha Harding 


Elise Cappella 
Sandee McClowry 


New York University 


Social-Emotional Learning (SEL) programs are school-based preventive interventions that aim to improve childrens social— 
emotional skills and behavioral development. Although meta-analytic research has shown that SEL programs can improve 
academic and behavioral outcomes in the short term, few studies have examined program effects on receipt of special educa- 
tion services and grade retention in the longer term. Using an experimental design, the current study leveraged administrative 
data available through students’ school records (N = 1,634) to examine the impacts of one SEL program implemented in 
kindergarten and first grade on receipt of special education and grade retention in fifth grade. The study further considered 
whether impacts varied for low- versus high-income students. Findings revealed no difference between treatment and control 
group students in grade retention. However, treatment group students were less likely to ever receive special education ser- 
vices by the end of fifth grade, with low-income students appearing to drive this effect. Implications are discussed. 


Keywords: 


Social-Emotional Learning (SEL) programs are school- 
based preventive interventions that aim to improve children’s 
social-emotional skills and behavioral development (Jones 
& Doolittle, 2017). Meta-analytic research has shown that 
SEL programs tend to achieve these aims in the short term 
(Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011). 
Further work (e.g., Bierman et al., 2010; Brackett, Rivers, 
Reyes, & Salovey, 2012; Diamond & Lee, 2011) demon- 
strates that SEL programs can improve children’s academic 
skills, although that evidence base is more mixed (Durlak 
et al., 2011; Social and Character Development Research 
Consortium, 2010). Few studies, however, have been able to 
track intervention study participants across time in order to 
understand the long-term impacts of SEL programming. 
Indeed, a recent meta-analysis by Taylor, Oberle, Durlak, and 
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Weissberg (2017) found that of 82 SEL program impact stud- 
ies with at least 6 months of follow-up data, only 6 studies 
considered effects 4 or more years postintervention. 

To date, evaluators have generally tested how SEL pro- 
grams affect children’s developmental outcomes, perhaps 
in line with theory hypothesizing that improvements in 
skills resulting from intervention will beget gains in later 
skills (Heckman, 2006; Magnuson & Duncan, 2016). A 
subset of rigorous evaluations has examined effects of 
SEL programs on receipt of special education (SPED) 
services and grade retention (Taylor et al., 2017), two out- 
comes that policy makers are increasingly interested in 
due to their cost implications for districts. In addition, 
recent work has further theorized that receipt of SPED 
services and grade retention may be particularly 
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important outcomes for determining children’s future 
educational trajectories. For example, as theorized by 
Bailey, Duncan, Odgers, and Yu (2017), some interven- 
tions can have long-term outcomes because they may 
“equip a child with the right skills or capacities at the 
right time to avoid imminent risks (. . .) or seize emerging 
opportunities” (p. 7). In general, SEL programs aim to 
change student behaviors by supporting children in recog- 
nizing/managing emotions, setting and achieving goals, 
appreciating the perspectives of others, establishing and 
maintaining positive relationships, making good deci- 
sions, and handling interpersonal situations construc- 
tively (Elias et al., 1997). In changing these behaviors, it 
is also possible that SEL programs may qualitatively shift 
the likelihood that children would be referred to receive 
SPED services or be retained in grade because behavioral 
issues would be less likely confounded with need for 
SPED services or grade repetition. Yet, few studies have 
examined effects of SEL programs on receipt of SPED 
services and grade retention partly because longitudinal 
follow-up on a larger sample is needed to detect impacts 
on these outcomes. 

It is also critical to consider variation in impacts for the 
children most likely to benefit from SEL programs in the 
longer term. Work rooted in prevention science asserts that 
children at heightened risk for social-emotional and/or 
academic difficulties may exhibit the largest improvements 
in outcomes after participating in an SEL program 
(Cicchetti & Aber, 1998; Jones, Brown, & Lawrence Aber, 
2011; Rimm-Kaufman et al., 2014). At the same time, theo- 
ries of differential susceptibility assert that some children 
are more likely to be affected by environmental contexts 
and supports than others (Belsky, Bakermans-Kranenburg, 
& Van Ijzendoorn, 2007). Early behavioral issues are asso- 
ciated with heightened risk for SPED services and/or grade 
repetition (Blair, 2001; Xia & Kirby, 2009). Given that a 
much higher proportion of children from low-income back- 
grounds are reported to have behavioral problems (Qi & 
Kaiser, 2003), and that children with behavioral and aca- 
demic problems may demonstrate the greatest gains from 
SEL programming (Rimm-Kaufman et al., 2014), it is 
plausible that low-income children who participate in SEL 
interventions will be less likely to receive SPED services 
and be retained in grade, in particular. Yet, few longitudinal 
studies have had diverse enough samples to examine how 
impacts of SEL programs vary based on individual charac- 
teristics and experiences. 

Leveraging longitudinal data from a randomized control 
trial of one SEL program implemented at the transition to ele- 
mentary school—/NSIGHTS into Children’s Temperament— 
we add to the literature by examining intervention impacts on 
grade retention and receipt of SPED services at the end of fifth 
grade. We then test for variation in program impacts for low- 
versus high-income children. 


Effects of SEL Programs on Grade Retention and Receipt 
of Special Education 


The transition to formal schooling—defined in this article 
as the start of kindergarten—presents multiple social 
demands, such as exhibiting nonaggressive and well- 
regulated behavior (Thompson & Raikes, 2007), paying 
attention and mastering core academic skills (Eisenberg, 
Valiente, & Eggum, 2010), and partaking in social interac- 
tions (Kellam et al., 2008). Participation in SEL programs 
can support young children to develop core skills of self- 
management, self-awareness, social awareness, responsible 
decision making, and relationship skills (Collaborative for 
Academic, Social, and Emotional Learning, 2012), which in 
turn should bolster children’s ability to successfully engage 
in these behaviors and navigate the transition to formal 
schooling. This is an important foundation because when 
students do not develop critical social and behavioral readi- 
ness skills at the transition to elementary school, the implica- 
tions for schools can be even more costly with regard to 
greater likelihood of grade retention, receipt of academic 
support services, and suspensions/expulsions (Bettencourt, 
Gross, & Ho, 2016). 

Evidence-based SEL programs differ in their 
approaches, but tend to use teacher professional develop- 
ment (e.g., training sometimes paired with coaching) and 
a classroom-based curriculum (which may vary in the 
extent to which they are scripted) to support teachers in 
using strategies that help students develop social—emo- 
tional skills (Low, Smolkowski, & Cook, 2016). Teachers 
are generally the primary delivery mechanism, often inte- 
grating SEL content throughout instructional time in addi- 
tion to program components that are more manualized. 
Teachers may support students to recognize and respond 
to different emotions (Preschool PATHS; Bierman et al., 
2010), use evidence-based strategies to respond to chal- 
lenging behaviors (Incredible Years; Webster-Stratton, 
Jamila Reid, & Stoolmiller, 2008), or use information on 
children’s unique temperaments to apply differentiated 
sets of strategies for responding to individual child behav- 
iors (i.e., INSIGHTS; McClowry, 2014). 

There is robust evidence demonstrating that children who 
participate in SEL programs exhibit fewer disruptive behav- 
ior problems and better behavioral regulation postinterven- 
tion than children in a comparison or control group (e.g., 
Durlak et al., 2011; Taylor et al., 2017). Developmental cas- 
cades theory would suggest that interventions like SEL pro- 
grams implemented at a sensitive time like the transition to 
kindergarten (Heckman, 2006; Masten, Long, Kuo, 
McCormick, & Desjardins, 2009; Reynolds & Temple, 2006) 
can lead to positive cascading effects by targeting compe- 
tence in domains that are linked to functioning across other 
domains over time (Masten & Cicchetti, 2010). For instance, 
SEL interventions can trigger such a cascade by targeting 
social-emotional competencies (e.g., self-regulation) which 


are linked to improved functioning across multiple domains 
(e.g., behavior, academic skills; Durlak et al., 2011). Given 
the increased likelihood for teachers to recommend children 
with behavioral and emotion regulation problems for reten- 
tion or SPED services (Bettencourt et al., 2016; Davoudzadeh, 
McTernan, & Grimm, 2015), by supporting student’ social— 
emotional competencies, SEL interventions may lead to a 
lower likelihood of referral to SPED or grade retention due to 
the positive cascading effects triggered by the intervention. 


Receipt of Special Education Services 


Schools generally begin providing SPED services to stu- 
dents in kindergarten (Levine, 2016). There are a variety of 
reasons why children might be referred to services and given 
an Individualized Education Plan, but disruptive behavior 
problems are one key reason for SPED referral at the transi- 
tion to formal schooling (Carlson et al., 2009; McIntyre, 
Eckert, Fiese, Reed, & Wildenger, 2010). Low-income stu- 
dents are also more likely than wealthier students to receive 
SPED services (either via pull out from the classroom, refer- 
ral toa SPED contained classroom, or enrollment at a school 
specifically for SPED students) due to behavioral issues 
(Coutinho & Oswald, 2000; C. O’Connor & Fernandez, 
2006). Although SPED services are critical to supporting the 
positive development of children who need them (Morgan & 
Sideridis, 2006), some children are referred to SPED simply 
due to difficulties in supporting their early behavioral regu- 
lation (Morgan et al., 2015). SPED services might be less 
relevant for these students because they mostly require 
behavioral supports in the classroom during early schooling, 
rather than a more comprehensive set of services across the 
full elementary school experience. 

Although children who receive SPED services do for the 
most part require intervention supports, it is also true that 
these children are more likely to exhibit poor academic and 
social-emotional outcomes in middle childhood and early 
adolescence, and are more likely to stay in SPED throughout 
schooling rather than be integrated in general education class- 
rooms (Harry & Klingner, 2014). SPED services are also 
costly: The Individuals with Disabilities Education Act allo- 
cated approximately $4,700 per student receiving SPED ser- 
vices for the 2013-2014 academic year (Snyder, de Brey, & 
Dillow, 2016), over and above the typical cost of general 
education. 

There is some empirical evidence to suggest that SEL 
programs implemented in early elementary school can 
reduce receipt of SPED services. In their long-term follow- 
up of the Good Behavior Game paired with an academically 
enhanced curriculum implemented in first grade, Bradshaw, 
Zmuda, Kellam, and Ialongo (2009) found that low-income 
African American children assigned to the intervention were 
6 percentage points less likely to receive SPED services 11 
years postintervention, relative to the control group. Even a 
small reduction in the percentage of students receiving 
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SPED services can represent a likely cost savings for 
districts. 


Grade Retention 


Ina similar way, participation in SEL programming could 
change whether children are retained in grade. Although 
some studies have found positive short-term impacts of 
grade retention on behavioral and psychosocial outcomes 
(e.g., Wu, West, & Hughes, 2010), the majority of work con- 
sidering longer term follow-up periods has generally found 
grade retention to have negative effects on students (Hong & 
Yu, 2007; Hwang & Cappella, 2018). For instance, using a 
regression discontinuity design based on a test score cutoff, 
Mariano, Martorell, and Tsai (2018) found that grade reten- 
tion reduced high school credit accumulation and the likeli- 
hood of taking math and English Regents exams. In addition, 
Hughes, West, Kim, and Bauer (2018) used a propensity 
score analysis and found that retention in Grades | to 5 
increased the likelihood of high school dropout. Similar to 
SPED services, grade retention in itself is expensive. 
According to a conservative estimate of a 1% annual reten- 
tion rate, nearly five hundred thousand children are retained 
in the United States each year (Eide & Goldhaber, 2005; 
Warren & Saliba, 2012). Given that in 2013-2014, the aver- 
age per pupil cost of an additional year in public school 
amounted to $12,509 ($24,116 in New York; U.S. Department 
of Education, National Center for Education Statistics, 
2017), the estimated annual cost of retention is approxi- 
mately $6.6 billion per year. 

When children display enhanced social-emotional skills 
in the classroom as a result of SEL intervention, it is possible 
that these enhanced competencies will reduce the likelihood 
that they will be retained in grade due to behavioral issues. 
Indeed, recent work by Mattison et al. (2018) has found that 
kindergarten students whose teachers rate them to be high on 
externalizing behavior problems are more likely to be 
retained between first and fifth grades. Some quasi- 
experimental work has found that participation in SEL pro- 
grams can reduce the probability that students will be retained 
in grade. For example, Hawkins, Catalano, Kosterman, 
Abbott, and Hill (1999) found that children who received the 
Seattle Social Development program (a combination of 
teacher training, parenting classes, and social competence 
training for children in Grades 1—6) were 8 percentage points 
less likely to repeat a grade by age 18 years, relative to a 
matched (but not randomly assigned) control group. 

However, the number of evaluation studies that have con- 
sidered receipt of SPED services and grade retention as out- 
comes are limited. In addition, the extant work in this domain 
examines interventions implemented in the early 1980s and 
1990s, prior to the zeitgeist movements in the early 2000s to 
not only expand the availability and quality of SEL programs 
but also to test their efficacy using rigorous randomized tri- 
als. Questions thus remain about whether the field is likely 


3 


McCormick et al. 


to see long-term effects of contemporary SEL programs on 
receipt of SPED services and grade retention. 


Variation in Impacts on Receipt of SPED for Low-Income 
Children 


SEL programs may have larger impacts on the SPED 
receipt and grade retention of lower income students. Indeed, 
lower income children are more likely than affluent students 
to receive SPED for behavioral issues in the early grades 
(Harry & Klingner, 2014), a core targeted outcome of SEL 
programming. This overrepresentation may be a function of 
lower income children exhibiting more behavior problems, 
or else may indicate teachers’ heightened frustration with 
behavior problems in classrooms and schools serving primar- 
ily lower income children (Bettencourt et al., 2016). Similarly, 
whereas grade retention can actually be beneficial for higher 
income children perhaps due to redshirting practices and/or 
because teachers in schools with more resources are better 
attuned to the needs of these students and observe benefits in 
retention (Fortner & Jenkins, 2017), additional research finds 
that lower income and racial/ethnic minority students are at 
heightened risk for grade retention due to early behavior 
problems (Davoudzadeh et al., 2015). If an SEL program 
does succeed in achieving its target of improving students’ 
behavioral regulation, this may be more effective in reducing 
receipt of SPED and grade retention for lower income stu- 
dents who are at heightened risk for these outcomes specifi- 
cally because of early behavior issues at the transition to 
elementary school. Few studies to date, however, have had 
diverse enough samples to explicitly test this hypothesis. 


INSIGHTS Jnto Children’s Temperament 


INSIGHTS is a comprehensive SEL program that sup- 
ports children’s ability to self-regulate by enhancing their 
attention and behavior management. Details about the inter- 
vention components are included in Appendix A. A prior 
study found that the program reduced children’s disruptive 
behavior problems and improved sustained attention by the 
end of first grade (E. E. O’Connor, Cappella, McCormick, & 
McClowry, 2014). This study did not consider program 
effects on SPED receipt and grade retention because the 
follow-up was not long enough to observe sufficient change 
in these outcomes to detect statistically significant program 
impacts. Indeed, in experimental studies of SEL programs it 
is rare to have longitudinal data to examine outcomes many 
years after the end of the intervention. Longitudinal data are 
likely necessary when estimating impacts on receipt of 
SPED services and grade retention because the changes in 
these outcomes take time to manifest themselves. However, 
administrative data on these outcomes were available as 
children who participated in study schools moved from kin- 
dergarten through fifth grade, making it possible to estimate 


impacts on receipt of SPED and grade retention in a longer 
term follow-up study. 

Although there are other examples of prior studies that con- 
sidered long-term effects of SEL programs on receipt of SPED 
services (Bradshaw et al., 2009) and grade retention (Hawkins 
et al., 1999), the current effort adds to the extant literature in 
four key ways. First, the prior experimental and quasi-experi- 
mental studies that examined long-term effects on receipt of 
SPED (Bradshaw et al., 2009) and grade retention (Hawkins 
et al., 1999) considered interventions that were implemented in 
the early 1990s. Given the proliferation of SEL programs and 
practices in schools across the past 15 years, the studies from 
the 1990s likely generated a much larger service contrast in 
provision of SEL activities than would be expected today. 
More up to date analyses may provide more realistic impact 
estimates for current programs operating today. 

Second, both the Bradshaw et al. (2009) and Hawkins 
et al. (1999) studies were conducted in within-group, high- 
need samples. Accordingly, the generalizability of these 
findings may be limited to the populations and contexts 
under which they were examined. In contrast, the current 
study adds to the literature because it includes information 
on a sample from a different city and contains sufficient 
diversity in the student population to consider variation in 
impacts by students’ family income. Third, the interventions 
studied in prior work were fairly intensive and required mul- 
tiple days of time from teachers to be trained on the interven- 
tions, and a larger amount of resources to support 
implementation. The JVS/GHTS intervention, in contrast, is 
comparatively easier to implement and requires less inten- 
sive commitment from teachers, students, and schools. 
Accordingly, the current study adds to the literature by esti- 
mating impacts of a less intensive SEL program (perhaps 
more scalable in future work) for potentially reducing SPED 
receipt and grade retention. 

Finally, more generally, given the documented difficulty in 
replicating effects across intervention studies noted by Stroebe 
and Strack (2014) and described by many others in the social 
sciences as the “replicability crisis,” it is critical to consider 
replicating impacts across different interventions and across 
time. Examining similar sets of outcomes across different 
intervention studies is important for building the evidence base 
on the range of curricula and SEL programs that can have last- 
ing impacts on SPED receipt and grade retention. The current 
study is well-poised to add to the literature by determining 
whether different types of models—including less intensive 
SEL programs—have impacts on receipt of SPED and grade 
retention, testing effects in more contemporary data, and 
examining variation in impacts using longitudinal data. 


The Current Study 


To this end, the current article will first estimate the 
impacts of one SEL program for kindergarten and first-grade 


students—JNSIGHTS into Children’s Temperament—on 
receipt of SPED services and grade retention at the end of 
fifth grade. Second, the article will consider how impacts of 
INSIGHTS vary for children from lower income families. 
This study will build evidence about how one particular SEL 
program does or does not affect the extent to which children 
access different educational experiences that they otherwise 
would not have been exposed to in the absence of the 
intervention. 


Method 


Between 2008 and 2012, a total of 22 elementary schools 
from three New York City school districts were randomly 
assigned to participate in the JVS/GHTS intervention or to an 
attention-control condition. The current study uses adminis- 
trative data for the students enrolled in study schools in the 
fall of kindergarten, which was the point at which schools 
were randomly assigned to study conditions. The study uses 
administrative data on children’s demographic characteris- 
tics, subsequent school locations, receipt of SPED, and 
grade retention between kindergarten and fifth grade. 


Participants and Setting 


Participants in this study included all students (NV = 1,634) 
who were enrolled in kindergarten—the time of random 
assignment—in an intervention or control school during the 
years when the randomized controlled trial took place.' See 
demographic characteristics of the study schools and stu- 
dents in Table 1. Children included in the current study sam- 
ple were representative of the broader total school 
population.” 


Recruitment and Randomization 


Schools were initially recruited for participation in the 
intervention study between 2008 and 2010. The recruitment 
strategies were approved by university and school system 
research boards. Principals serving primarily low-income 
students in three urban school districts were the first to be 
contacted. Team members explained the purpose of the 
study and its related logistics including randomization to one 
of two intervention conditions: INSIGHTS or the attention- 
control condition which was an after-school reading pro- 
gram (see further details below). Twenty-three schools were 
invited and initially agreed to participate. Prior to random- 
ization, one school withdrew from the study due to a princi- 
pal transition. Teachers were initially recruited in small 
group or individual meetings. Ninety-six percent of the kin- 
dergarten and first-grade teachers at participating schools 
consented to participate. All teachers agreed to participate 
for the duration of the study. 

In the original study, a subsample of students within the 
participating schools (NV = 435) received parental consent for 
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data collection (see E. E. O’Connor et al., 2014, for more 
details). In contrast, the current study leverages de-identified 
data on all students who were enrolled in participating 
schools and thus represents a larger sample than the group of 
students who consented for research activities. This is pos- 
sible because all children in participating schools received 
the intervention delivered by teachers and _ facilitators, 
regardless of whether they had consented to participate in 
data collection. The only difference for the consented sub- 
sample is that parents in INSIGHTS schools received the 
opportunity to participate in the parent program. In the cur- 
rent study, students were identified as members of the treat- 
ment group based on their enrollment in kindergarten—the 
point or random assignment—in one of the schools assigned 
to INSIGHTS. 

After baseline data were collected in kindergarten, a ran- 
dom numbers table was used to randomize schools to 
INSIGHTS or the attention-control group. Schools were used 
as the unit of random assignment to limit possible contami- 
nation effects that could threaten the internal validity of the 
study. Eleven schools were randomized to INSIGHTS, and 
11 schools were assigned to the attention-control group. 
There were the same number of treatment and control group 
schools within each participating district. Chi-square tests 
were used to examine baseline equivalence between the 
INSIGHTS (N = 800 students, 11 schools) and attention-con- 
trol groups (NV = 843 students, 11 schools) on observed char- 
acteristics available on study schools and students enrolled 
in participating schools. We used independent samples ¢ tests 
to compare baseline characteristics between students and 
schools assigned to INSIGHTS versus the attention-control 
group. Similar to findings in E. E. O’Connor et al. (2014), 
preliminary analyses revealed no statistically significant dif- 
ferences between schools assigned to the JNS/GHTS pro- 
gram versus schools assigned to the attention-control group. 
Similarly, there were no differences between the group of 
students assigned to the JNS/GHTS group versus the atten- 
tion-control group. See Table 1 for more detailed informa- 
tion establishing baseline equivalence. 


Data Collection 


The data for this study were obtained from the New York 
City Department of Education through the Research Alliance 
for New York City Schools. The New York City Department 
of Education collects and records administrative data on all 
students at the beginning (October) and end (June) of each 
school year. The research team received administrative data 
through fifth grade for all students who were enrolled in kin- 
dergarten in one of the schools participating in the study. 


Measures 


Receipt of Special Education Services. In the administrative 
data, receipt of SPED services (excluding services related to 


TABLE | 
Baseline Descriptive Statistic for INSIGHTS and Attention-Control Groups 


Attention-control INSIGHTS 

Characteristic M SD M SD Difference 

School characteristics 
Annual attendance rate 88% — 86% — 0.02 
Female 51% = 49% — 0.02 
Black 76% — 79% — —0.03 
Hispanic 39% = 43% _ —0.04 
White 11% — 9% — 0.02 
Other race 13% — 12% _ 0.01 
Eligible for free or reduced-price lunch 72% — 75% — —0.03 
Dual language learner 8% — 6% — 0.02 
From recent immigrant family 6% = 5% = 0.01 
Prekindergarten attender 59% _ 62% — —0.03 
% students scoring average on ELA state test 53% — 49% — 0.04 
% students scoring average on math state test 63% — 60% — 0.03 
Number of students in school 505.07 190.09 493.21 138.62 11.86 
Sample size 11 11 

Student characteristics for full sample 
Baseline age (years) 5.17 0.72 5.19 0.66 —0.02 
Female 48% = 48% = 0 
Black 73% — 717% — —0.04 
Hispanic 35% = 38% = —0.03 
White 12% — 10% — 0.02 
Other race 15% = 13% = 0.02 
Eligibility for free or reduced price lunch 72% _ 75% — —0.03 
Dual language learner 9% — 11% — —0.02 
From recent immigrant family 4% = 6% — —0.02 
Prekindergarten attender 62% = 66% = —0.04 
Received special education (SPED) services at baseline 1% — 1% — 0 
Sample size 843 800 

Student characteristics for nonattrited analytic sample 
Baseline age (years) 5.24 0.71 5.18 0.69 0.06 
Female 49% = 50% — —0.01 
Black 75% — 79% — —0.04 
Hispanic 36% _ 39% — —0.03 
White 11% — 9% — 0.02 
Other race 14% _ 12% — 0.02 
Eligibility for free or reduced price lunch 77% — 79% _ —0.02 
Dual language learner 10% — 9% — 0.01 
From recent immigrant family 5% — 6% _ —0.01 
Prekindergarten attender 65% _ 69% — —0.04 
Received SPED services at baseline 1% — 1% — 0 
Sample size 639 624 


Note. We used independent-samples ¢ tests to examine whether the treatment and control groups were significantly different from one another. None of the 
group-based differences in the final column were statistically significant. 


physical disabilities, vision, and/or hearing problems) was not provide details on the reason for receipt of SPED. 
indicated for each grade level for each study participant. Accordingly, receipt of SPED services could be related to a 
Receipt of SPED also included enrollment in a stand-alone _ host of issues (e.g., behavioral, academic, speech supports, 
SPED school. One limitation of these data is that they did __ etc.). We do know, however, that receipt of any language 


support services for English language learners is not included 
in this variable. In addition, receipt of SPED was highly cor- 
related across years, such that students who received SPED 
services in first, second, third, or fourth grades tended to 
continue to receive SPED services in each year thereafter. 

We first created grade-specific dummy variables to 
describe whether the student received SPED services within 
that grade. Once a variable was created for each grade, 
another dummy variable was coded to identify students who 
received SPED services during any grade from first grade 
(the year that intervention ended) to fifth grade, for example, 
ever referred for SPED services (1 = received SPED services 
between first and fifth grade; 0 = never received SPED ser- 
vices between first and fifth grade). 


Grade Retention. A dummy variable was created to indi- 
cate whether a student was ever retained between kinder- 
garten and fifth grade (1 = retained anytime between 
kindergarten and fifth grade; 0 = never retained). If the stu- 
dent’s actual grade level was behind the expected grade 
level in any grade between kindergarten and fifth grade, 
then the student was coded as retained. For the small num- 
ber of cases where values appeared problematic (e.g., if the 
expected grade level was behind in one year but then ahead 
of expected grade in a future year), we cross-referenced val- 
ues with the school discharge variable to determine grade 
retention categorization. 


Demographic Characteristics. At public school enrollment, 
parents and guardians reported on their children’s demo- 
graphic characteristics—race, ethnicity, gender, eligibility 
for free or reduced price lunch (used in the current study to 
describe a student as low-income), immigration status, birth- 
date, and whether their child spoke a language other than 
English. These variables were used as covariates in the anal- 
ysis. Child age on September | of the kindergarten year was 
also calculated using the birthdate and included as a covari- 
ate in analytic models. 


School-Level Characteristics. We included a set of covari- 
ates at the school level as well in our predictive models that 
match the set of school-level covariates we used in our short- 
term follow-up study. We used publicly available adminis- 
trative data to capture the school size (number of students 
enrolled in the year prior to the intervention implementa- 
tion), school attendance rate, the percentage of students in 
the school who were proficient on the state ELA test in the 
prior year, and the percentage of students in the school pro- 
ficient on the state math test in the prior year. We further 
adjusted for study cohort at the school level including 
dummy variables for Cohort 2 and Cohort 3, with Cohort 1 
as the reference group. We used some additional school- 
level demographic characteristics to examine baseline 
equivalence at the school level. These were school-level 
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versions of the student-level demographic characteristics 
(percent female, percent Black, percent Hispanic, percent 
eligible for free/reduced price lunch, percent immigrant, 
percent dual language learner, and percent who attended a 
district PreK program). 


INSIGHTS Jntervention Procedures and Attention-Control 


Participating schools were divided into three cohorts 
(Cohort 1: Fall 2008-Spring 2010; Cohort 2: Fall 
2009-Spring 2011; Cohort 3: Fall 2010-Spring 2012) and 
agreed to implement the JNS/GHTS or attention-control pro- 
gram in kindergarten and first grade. More specific informa- 
tion about the intervention and attention-control conditions 
is located in Appendix A. 


Analytic Approach 


Missing Data Analysis. As noted above, the total analytic 
sample size for this study is N = 1,634 students. Of the total 
number of students, 1,263 (77%) remained in the sample 
through the end of the fifth grade, while 371 (23%) are miss- 
ing outcome data because they attrited from the school dis- 
trict before the end of the fifth grade. Follow-up analyses 
revealed that students in the treatment and attention-control 
group attrited at similar rates (24% of the baseline attention- 
control group attrited and 22% of the treatment group 
attrited). In Table 1, we further illustrate how the baseline 
characteristics are similar across the nonattrited students in 
the treatment and attention-control groups and further detail 
on attrition is included in Appendix B. According to the 
standards from the What Works Clearinghouse (2017), this 
study constitutes a low-attrition RCT, meaning that overall 
attrition and differential attrition indicate a tolerable level of 
bias for the estimated intervention effect. Using the assump- 
tion that data were Missing at Random, the team used a mul- 
tiple data imputation method in STATA MICE and imputed 
100 data sets in order to generate complete data on all covari- 
ates, which had minimal levels of missingness (Graham, 
Olchowski, & Gilreath, 2007). 


Impact Analysis. Because school was the original unit of 
random assignment and most students stayed in the same 
elementary school that they enrolled in during kindergarten, 
we expected that student outcomes would not be indepen- 
dent at follow-up. In order to determine the most appropriate 
fit to the data, we considered a number of different models 
and compared their log likelihoods and fit statistics (Akaike 
information criterion [AIC] and Bayesian information crite- 
rion [BIC]). This approach for model selection has been rec- 
ommended by Scott, Simonoff, and Marx (2013). For both 
sets of outcomes—receipt of SPED services and grade reten- 
tion through fifth grade—we first ran unconditional logit 
models with clustered standard errors for schools. Then, we 


TABLE 2 


Descriptive Statistics for Grade Retention and Special Education (SPED) Referral Through Fifth Grade Unadjusted 


Outcome variable INSIGHTS Attention-control Difference Significant difference 
Full sample 

Ever retained in grade 25.86% 24.61% 1.25% 

Ever received SPED services 6.95% 10.34% —3.39% * 
Low-income sample 

Ever retained in grade 22.89% 21.79% 1.10% 

Ever received SPED services 8.87% 13.46% —4.59% * 
Higher income sample 

Ever retained in grade 10.43% 9.52% 0.91% 

Ever received SPED services 6.86% 5.80% 1.06% 


Note. N= 1,263 students with available outcome data, N = 973 students in the low-income group, N = 290 in the high-income group. 


*p <.05. **p < 01. 


compared that model fit to a mixed-effects logistic regres- 
sion (Fitzmaurice, Laird, & Ware, 2004) which included ran- 
dom effects for the school that participated in the original 
trial. After comparing the log likelihoods of the models and 
the AIC/BIC statistics across these specifications (these sta- 
tistics are illustrated in Model | in Tables 3 and 4 for the 
mixed-effects logistic regressions and in Appendix D Tables 
D1 and D2 for the logistic regressions with clustered stan- 
dard errors for study school), we found that the mixed- 
effects logistic regression was a better fit to the data for both 
outcomes. We then considered a number of covariance struc- 
tures, including the unstructured, independent, exchange- 
able, and identity structures. Results were almost identical 
regardless across approaches, so we retained the unstruc- 
tured covariance structure for our main set of analyses. The 
baseline random intercepts model for examining treatment 
impact is illustrated in Equation 1. 


logit {Pr(y, = LX. Gi) = B, + BJNSIGHTS; + 6; (1) 


where yy, refers to the outcome for student 7 in school /; C_ is 
a random intercept for school, and B, i is the treatment impact 
for students who participated in the study in school j. It is 
important to note here that assignment to JNS/GHTS is oper- 
ationalized as a school-level variable. The full set of nested 
models including all Level-1 and Level-2 covariates is 
included in Appendix C. After estimating the main effects of 
INSIGHTS on the study outcomes, we then considered how 
impacts varied for low-income students by creating cross- 
level interactions between assigned to INSIGHTS and the 
dummy variable for low income. 

All findings are reported as odds ratios. For statistically sig- 
nificant main effects and moderated impacts, we then calcu- 
lated probabilities for treatment and control group members. 
Differences in the probabilities can be interpreted as percent- 
age differences and facilitate interpretation and comparisons 
with prior research. Given the relatively small number of 


schools in the study and potential for imbalance in findings, we 
also conducted a series of robustness checks and tested models 
using logistic regression with clustered standard errors for 
study school. We further considered how robust the impacts 
were when we group-mean centered the Level-1 covariates. 


Results 
Descriptive Statistics for Outcome Variables 


Frequencies for the overall sample, and for both low- 
income and higher income groups by intervention status are 
reported in Table 2. Independent-samples ¢ tests revealed that 
students in the JVS/GHTS group were significantly less likely 
to receive SPED services through fifth grade: mean difference 
= —3.39; ¢(1,261) = 14.56, p < .05. In addition, low-income 
children enrolled in the JVS/GHTS group were also less likely 
to receive SPED services through the end of fifth grade relative 
to low-income children enrolled in the attention-control condi- 
tion: mean difference = —4.59; t(1,261) = 24.86, p < .05. 


Impacts of INSIGHTS on Grade Retention and Receipt of 
SPED Services 


Impact results examining main effects are presented in 
Table 3 (grade retention) and Table 4 (receipt of SPED ser- 
vices). As shown here, there were no main effects of 
INSIGHTS on grade retention through fifth grade (odds ratio 
[OR] = 0.79, p = .33). However, there was a statistically sig- 
nificant impact on receipt of SPED services such that stu- 
dents enrolled in INSIGHTS schools were less likely to 
receive SPED services through fifth grade than students in 
the attention-control condition (OR = 0.85, p = .04). We used 
the conditional odds ratio to calculate the probability of 
receiving SPED services by fifth grade for a student in the 
INSIGHTS group versus the control group, who was a mem- 
ber of the reference group on all covariates. Reference group 
students in the JVSIGHTS group had a 7% probability of 


TABLE 3 
Impacts of INSIGHTS on Grade Retention 


Model 1 Model 2 Model 3 Model 4 
Fixed effects OR Log(odds) SE OR Log(odds) SE OR Log(odds) SE OR _ Log(odds) SE 
Intercept 0.27 -1L.31** O15 0.19 -1.67** 0.21 0.33 —-1.10** 0.24 0.34 —-1.08** 0.22 
Treatment (INSIGHTS) 0.77 —0.26 0.29 0.79 —0.23 0.22 0.79 -0.24 0.19 
Student-level covariates 
Baseline age 0.61 —0.49** O11 0.63 —0.46** 0.09 
Female 0.82 —0.20 0.15 0.79 -0.24 0.16 
Black 1.07 0.07 0.36 1.01 0.01 0.35 
Hispanic 1.31 0.27 0.49 1.32 0.28 0.49 
Low SES 2.01 0.70* 0.33 2.04 0.71* 0.31 
Dual language learner 0.63 —0.46* 0.19 0.59 —-0.53* 0.17 
From recent immigrant 0.72 —-0.33 0.27 0.75 —0.29 0.29 
family 
Prekindergarten attender 0.61 —0.49** 0.11 0.58 —0.54** 0.09 


School-level covariates 


Cohort 2 
Cohort 3 


1.21 0.19 0.25 
1.22 0.20 0.23 


School size 
School % competent 


1.33 0.29 0.28 
0.87 —0.14* 0.06 


ELA test 
School % competent 


0.81 —0.21* 0.05 


math test 
School average 


0.85 -—0.16 0.15 


attendance 


Variance components Variance SE 


Variance SE 


Variance SE Variance SE 


Baseline study school 0.08* 0.03 0.08* 0.04 0.06 0.04 0.06 0.04 
Fit statistics 

AIC 351.48 342.47 328.47 318.46 

BIC 358.89 362.57 351.74 334.42 

Log likelihood —173.64 —171.24 —167.84 —161.84 


Note. OR = odds ratio; SE = standard error; SES = socioeconomic status; AIC = Akaike information criterion; BIC = Bayesian information criterion. We 
used a series to models in this impact analysis to examine the sensitivity of our results to the inclusion of covariates. Model | in this table is an unconditional 
model that includes no predictors. Model 2 examines the impacts of INSIGHTS on the outcome adjusting for no covariates. Next, Model 3 tests how sensitive 
the impact results are to the inclusion of a block of student-level covariates. Finally, Model 4 further tests how sensitive the impact results are to the inclusion 


of blocks of both student- and school-level covariates. 
*p<.10. *p <.05. **p <.01. 


receiving SPED services by fifth grade, relative to a 12% 
probability for students in the attention-control condition. 
We found that null effects of INSIGHTS on grade reten- 
tion through fifth grade did not vary by student income. 
However, we did find a statistically significant interaction 
between assignment to INSIGHTS and low-income in pre- 
dicting receipt of SPED services through fifth grade (OR = 
0.39, p = .04; see Figure 1). The finding demonstrates that 
children who were low-income and in JNS/GHTS schools 
were less likely than low-income students attending schools 
in the attention-control group to receive SPED services 
between kindergarten and fifth grade. Using the odds ratio 
estimate, we calculated the probabilities for the two groups 


in the adjusted models, finding that low-income students in 
the INSIGHTS group (in the reference group for all covari- 
ates) had a 9% probability of receiving SPED services by 
fifth grade, relative to the low-income students in the control 
group who had a 15% probability of receiving SPED ser- 
vices by fifth grade. Further probing demonstrated that there 
was no difference in receipt of SPED for higher income stu- 
dents assigned to INSIGHTS versus the attention-control. 


Robustness Checks 


We used a series of robustness checks to examine how 
sensitive the impact analysis results were to our model 


TABLE 4 
Impacts of INSIGHTS on Receipt of Special Education Services 


Model 1 Model 2 Model 3 Model 4 

Fixed effects OR Log(odds) SE OR Log(odds) SE OR Log(odds) SE OR Log(odds) SE 
Intercept O10 =2.30** 0.23 0.09 —2.41** 0.01 0.03 —3.51** 0.02 0.01 —4.61** 0.01 
Treatment (INSIGHTS) 0.83 —0.19 0.12 0.81 =0.21* 0.09 0.85 —0.16* 0.07 
Student-level covariates 

Baseline age 1.12 0.11 0.24 1.18 0.17 0.22 

Female 0.44 —0.82** 0.14 0.42 =0:87** 0.08 

Black 0.17 =L77** 0.11. 0.13 —2.04** 0.18 

Hispanic 0.31 -1L.17** 0.21 0.34 —1.08** 0.08 

Low SES 0.28 =1.27* 0.56 0.24 —1.43** 0.56 

Dual language learner 2.01 0.70 0.42 = 1.95 0.67 0.38 

From recent immigrant family 0.63 —0.46 0.28 0.58 —0.54 0.31 

Prekindergarten attender 0.89 —0.12 0.16 0.82 —0.20 0.16 
School-level covariates 

Cohort 2 1.03 0.03 0.43 

Cohort 3 1.11 0.10 0.53 

School size 1.31 O27" 0.11 

School % competent ELA test 0.95 —0.05 0.21 

School % competent math test 0.91 —0.09 0.33 

School average attendance 1.11 0.10 0.29 
Variance components Variance SE Variance SE Variance SE Variance SE 
Study school 0.09** 0.03 0.09* 0.03 0.09* 0.02 0.06* 0.02 
Fit statistics 

AIC 210.65 206.65 178.56 165.86 

BIC 217.67 211.76 186.43 180.32 

Log likelihood —103.03 —101.91 —94.56 —93.45 


Note. OR = odds ratio; SE = standard error; SES = socioeconomic status; AIC = Akaike information criterion; BIC = Bayesian information criterion. We used a series of models 
in this impact analysis to examine the sensitivity of our results to the inclusion of covariates. Model | in this table is an unconditional model that includes no predictors. Model 2 
examines the impacts of JVS/GHTS on the outcome adjusting for no covariates. Next, Model 3 tests how sensitive the impact results are to the inclusion of a block of student-level 
covariates. Finally, Model 4 further tests how sensitive the impact results are to the inclusion of blocks of both student- and school-level covariates. 


“p< .10. *p <.05. **p <01. 
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15% 


Percent received SPED services by 
fifth grade 


0% esa 


Low-income students Higher-income students 


INSIGHTS Attention-control 
FIGURE 1. Impacts of INSIGHTS on receipt of SPED services 
by fifth grade. 


*p <.05. **p < 01. 


specification. In general, we found that the bulk of our results 
were robust across different modeling assumptions. All 
robustness checks are described thoroughly in Appendix D. 
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Discussion 


This study aimed to leverage existing administrative data 
on all students enrolled in schools that participated in a ran- 
domized control trial of the IVS7GHTS program to determine 
whether there were long-term effects of the intervention on 
receipt of SPED services and grade retention from kindergar- 
ten through fifth grade—and whether outcomes varied for 
lower income versus higher income students. Findings 
revealed that students in the INSIGHTS group were 5 per- 
centage points less likely than comparison group members to 
receive SPED services by fifth grade. However, this effect 
was sensitive to model selection and did not fully replicate in 
a model using clustered standard errors for study school. 
Even so, we further found that JVS7GHTS reduced the likeli- 
hood of receiving SPED services for /ow-income students by 
6 percentage points. This impact was not sensitive to model 
selection and we are thus more confident that the intervention 
reduced receipt of SPED services within the sample of low- 
income students. There were no main or moderated effects of 
INSIGHTS in reducing grade retention by fifth grade. 


Impacts of INSIGHTS on receipt of SPED services are 
somewhat aligned with prior research. For example, 
Bradshaw et al. (2009) found that assignment to the Good 
Behavior Game reduced the likelihood of placement in a 
SPED classroom by 6 percentage points with is a similar 
effect size to the one found in this study, although the fol- 
low-up period was about twice as long in that prior study, so 
the research designs are not directly comparable. Work by 
Rimm-Kaufman, Pianta, and Cox (2000) has found that kin- 
dergarten teachers working in schools serving primarily 
low-income families are more likely to report significant 
behavioral problems for children at kindergarten entry. 
Lower income students who exhibit behavior problems in 
early educational settings are more likely than higher income 
students with equivalent behaviors to be referred to SPED 
(C. O’Connor & Fernandez, 2006; Shavelson & Towne, 
2002). An earlier study demonstrated that INSIGHTS did 
reduce children’s disruptive behaviors in the short term and 
improved their literacy and math skills (O’Connor et al., 
2014). Importantly, children identified to be at behavioral 
risk at baseline benefited the most from the intervention in 
terms of their reduced disruptive behaviors and improved 
behavioral engagement (McCormick, O’Connor, Cappella, 
& McClowry, 2015). Positive impacts on students’ behav- 
iors in the short term may have later manifested in broader 
reductions in receipt of SPED services. This result may have 
been particularly true for the lower income students in the 
sample with a higher likelihood of receiving SPED services 
due to challenging behaviors (Qi & Kaiser, 2003; Xia & 
Kirby, 2009). 

In contrast, the grade retention finding for this study is 
not fully aligned with prior work evaluating impacts of SEL 
programs (e.g., Hawkins et al., 1999; Hawkins et al., 2009). 
For example, in their 6-year follow-up of the Seattle Social 
Development Program, Hawkins and colleagues (1999) 
found that the program reduced grade retention by 12.5 per- 
centage points. The null impact found in the current study 
may reflect the fact that although the strategies taught by the 
INSIGHTS program were intended to be integrated into typi- 
cal instruction during the school day, the activities were not 
directly tied to academic content. The intervention itself may 
not have been sufficiently intense to improve children’s aca- 
demic skills to the point where it would change their likeli- 
hood of being retained in grade. SEL programs explicitly 
target social-emotional skills, which are theorized to then 
support gains in academic skills (Durlak et al., 2011). 
However, for children who are risk of being retained in grade 
based on poor early academic skills, more intensive supports 
directly focused on academic instruction may be necessary 
over and above involvement in SEL programming. 

In addition, it is important to note that the comparison 
group in this study did receive a reading intervention. As 
such, we are comparing /VSIGHTS—an SEL program—to a 
more active control group participating in an academically 
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oriented set of services rather than a business as usual con- 
trol group. We originally decided to use this approach in 
order to ensure that any treatment effects we observed were 
related to the INSIGHTS SEL program and not the fact that 
treatment schools were simply receiving additional attention 
and professional development supports. The current study 
thus represents a relatively conservative test of the JNS/GHTS 
program and more stringently examines the effect of a pro- 
gram to support social-emotional development relative to a 
more academically oriented intervention. With respect to the 
null effect on grade retention, it might have been that the 
supports from the Read Aloud program were similarly effec- 
tive in supporting children’s grade promotion. Further work 
examining the effects of JVS/GHTS—and other SEL stud- 
ies—on grade retention may consider a less active control 
group in order to increase the treatment contrast and see how 
that change does or does not affect impacts on grade 
retention. 

Prior studies that have looked at the longer term effects of 
SEL programs on student outcomes, albeit from implemen- 
tation efforts several years ago, have found longer run effects 
through high school and beyond (Taylor et al., 2017). For 
example, earlier studies have found effects on outcomes 
such as high school dropout or graduation (Bradshaw et al., 
2009; Felner et al., 1993; Hawkins, Kosterman, Catalano, 
Hill, & Abbot, 2008), college attendance or completing a 
college degree (Bradshaw et al., 2009; Hawkins et al., 2008), 
safe sexual behaviors and STD diagnosis (Hill et al., 2014), 
criminal involvement or arrests (Eddy, Reid, Stoolmiller, & 
Fetrow, 2003; Hawkins et al., 2008), and adult mental health 
outcomes (Hawkins et al., 2008; JIalongo, Poduska, 
Werthamer, & Kellam, 2001; Riggs & Pentz, 2009). More 
contemporary evidence (e.g., longer term impacts of the 
Positive Action program; Duncan et al., 2017) has demon- 
strated benefits of universal SEL program in reducing behav- 
ior problems through eighth grade. Yet, no studies to date 
have explicitly tested whether impacts of SEL programming 
on receipt of SPED services did in fact lead to longer term 
outcomes in high school and beyond. This study represents 
an initial step in identifying an impact of one SEL program 
on receipt of SPED services through fifth grade. Even so, 
additional research in the future would be needed to explic- 
itly test whether reductions in receipt of SPED services in 
the short term would then link to long-term effects in high 
school and beyond. 


Limitations 


Although this study uses an experimental design and 
multiple years of follow-up data for a large sample, there are 
a number of limitations. First, the study sample is made up 
of the total population of students who were enrolled in kin- 
dergarten in schools participating in the study during their 
first year of study implementation. Although we know that 
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students who consented to the study participated at high 
rates, we cannot determine the exact level of dosage for the 
students included in this study. In order to use an intent-to- 
treat design in this study, we included students in the treat- 
ment group if they were enrolled in a school assigned to 
INSIGHTS during the kindergarten year when the interven- 
tion was initially implemented. If the child was enrolled in a 
control group school in kindergarten and then switched to 
INSIGHTS, they remained in the control group in order to 
maintain the internal validity of the intent to treat design, 
and vice versa. Review of students’ school locations, how- 
ever, suggests that crossovers from treatment to control 
group school were minimal. It was more likely that a student 
would leave an INSIGHTS school for a nonstudy school or 
vice versa. Even so, we are unable to describe the extent to 
which we observed noncompliance in this study for the full 
student sample. 

Although the administrative data are a core strength of 
the study, it is still true that understanding these data is com- 
plicated, particularly with respect to grade retention. It is 
possible that there is some measurement error in operation- 
alizing the outcome constructs of interest in this study. In 
order to limit the possibility of Type I error posed by running 
too many impact analyses, we restricted this article to con- 
sider four total impact models and two total outcomes. It is 
possible that if we used different operationalizations of the 
outcomes (e.g., comparing impacts on grade retention at dif- 
ferent follow-up time points) we could have found differen- 
tial effects of the treatment across time. Further work with 
greater statistical power to conduct multiple comparisons 
adjustments may consider probing these outcomes further. 

We also found that the results of the analysis did not com- 
pletely replicate across a model using random intercepts for 
study schools and a logistic regression using clustered stan- 
dard errors for study school (see Appendix D). However, the 
findings from the logistic regression were similar in direc- 
tion and magnitude and there is a fierce debate in the litera- 
ture about the conceptual difference between findings at the 
p < .05 and p < .10 levels (Cristea & Ioannidis, 2018). 
Moreover, work by Roodman, Nielsen, MacKinnon, and 
Webb (2019) has shown that clustered standard errors may 
be too conservative of an approach when there are few clus- 
ters of interest. We do interpret the main effects of INSIGHTS 
on receipt of SPED services with caution given lack of exact 
replication. However, we have more confident that 
INSIGHTS did reduce receipt of SPED services for low- 
income students within the sample—an effect that replicated 
across all models. 

Finally, we lack detailed and nuanced data on the poten- 
tial moderators and mechanisms explaining impacts of 
INSIGHTS on receipt of SPED and why those impacts were 
larger for lower income children. Furthermore, we were 
unable to demonstrate that the impacts on receipt of SPED 
were not accompanied by negative impacts in other domains 
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within the current study sample. Doing so would have 
allowed us to be more confident in our assertion that reduc- 
ing receipt of SPED services is a positive outcome, and not 
actually indicative of a situation where children who need 
SPED services are unable to access them. Our past work 
examining the short-term impacts of the program (E. E. 
O’Connor et al., 2014) demonstrated that IVSIGHTS reduced 
disruptive behaviors and improved behavioral engagement 
and that those effects were driven by students who had chal- 
lenging temperaments at baseline (McCormick et al., 2015). 
In addition, our team has completed a complementary set of 
analyses (McCormick, O’Connor, Cappella, & McClowry, 
2019) that examine the impacts of INSIGHTS on academic 
skills outcomes in third, fourth, and fifth grades for students 
who were in the original trial (V = 331 with complete data at 
follow-up). Although these results are still under review, we 
have found positive treatment impacts on language skills in 
third and fourth grade and no group-based differences on 
children’s math skills. This is only a subsample of the group 
of students examined in the current study, but findings do 
provide further evidence that a reduction in receipt of SPED 
services is not associated with negative outcomes for stu- 
dents in other domains. 


Conclusions and Implications 


Results from this study demonstrate that a time-limited 
and fairly intensive SEL intervention that supports teachers 
in effectively responding to and managing the behaviors of 
children implemented at the critical transition to elementary 
school can reduce receipt of SPED services through fifth 
grade for low-income students. The fields of intervention 
and education research are currently interested in identifying 
the mechanisms through which early intervention can affect 
longer term outcomes. Rather than following a linear pro- 
cess, the current study provides some early evidence for 
Bailey et al.’s (2017) contention that early programming can 
provide access to qualitatively different experiences for 
students—for example, by reducing the likelihood of that 
they will be referred to SPED services due to a behavioral 
issue—which may lead to longer term adaptive outcomes. 
Low-income children who need SPED services should con- 
tinue to receive them. But SEL programs may be able to sup- 
port children’s behavioral development early on in a way 
that can prevent them from perhaps unnecessarily receiving 
SPED programming and better promoting their academic 
development at the transition to elementary school. 

Policy makers can benefit from this work when consider- 
ing cost implications of early programming, and integrating 
evidence-based SEL intervention into the early grades. 
Although some whole-school reforms to support students’ 
social emotional skills can be quite costly, shorter embedded 
interventions—like /JNS/GHTS—can be less expensive. 
Moreover, if SEL programs can have even small effects on 


outcomes like receipt of SPED services, they can still have 
significant cost savings for school districts serving low- 
income students. Although this study did not include an in- 
depth cost effectiveness component, one could potentially 
balance the cost of INSIGHTS implementation—estimated 
at about $5,500/school for 2 years of programming (or about 
$50/child)—against the expected cost of a lower income stu- 
dent receiving SPED services over and above regular educa- 
tion (average of $4,700/year; Snyder et al., 2016). However, 
this is likely an overestimate in comparing costs as many of 
the expenses associated with the receipt of SPED are likely 
driven by a few very high-need children and the students 
who may benefit from intervention are likely lower need. 
Future work including a more in-depth cost study focused 
explicitly on the subsample of students who do not receive 
SPED services as a result of an SEL program is needed. 
Finally, the study provides initial evidence that practitioners 
can be trained to support a diverse set of students’ social— 
emotional development in such a way that can link to adap- 
tive outcomes even after those students transition out of 
trained teachers’ classrooms. Integrating evidence-based 
SEL programming into teacher training and professional 
development on a broader scale may thus be warranted. 


Appendix A 


Information on the INSIGHTS Intervention and Attention- 
Control Conditions 


INSIGHTS Into Children’s Temperament. INSIGHTS is a 
comprehensive SEL program that supports children’s ability 
to self-regulate by enhancing their attention and behavior 
management. The program aims to help teachers learn to 
recognize the consistent behavioral style that a child exhibits 
across settings as an expression of temperament. Although 
temperament is stable, adult responses can be adjusted to 
match each child’s unique temperament. The aim of this 
effort is to enhance goodness of fit, the consonance of a 
child’s particular temperament to the demands, expecta- 
tions, and opportunities of the environment (Thomas & 
Chess, 1977). By accepting and understanding a child’s par- 
ticular temperament, teachers are able to use scaffolding and 
stretching strategies to better support the child when tem- 
peramentally challenging situations are encountered (Denis- 
sen, van Aken, Penke, & Wood, 2013; Rothbart, Sheese, 
Rueda, & Posner, 2011). Children also learn how to identify 
their own temperaments. The component of /NS/GHTS 
directed at children focuses on expanding children’s self- 
regulation and social-emotional skills. Children are intro- 
duced to four puppets with different temperaments during 10 
weeks of classroom sessions. Utilizing the puppets, the chil- 
dren explore how some situations are easy for certain pup- 
pets, while others are more challenging. During the 
remaining weeks, the children work with the puppets to use 
problem-solving strategies when confronted with daily 
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dilemmas. This part of the intervention supports children in 
identifying strategies matched to their temperament to 
address the daily challenges they may experience in the 
classroom. 


INSIGHTS Jntervention Procedure. Participating schools 
were divided into three cohorts (Cohort 1: Fall 2008—Spring 
2010; Cohort 2: Fall 2009-Spring 2011; Cohort 3: Fall 
2010—Spring 2012) and agreed to implement the JVS/GHTS 
or attention-control program in kindergarten and first grade. 
The INSIGHTS program was delivered to teachers and chil- 
dren concurrently by facilitators. The goal of the model was 
that after the end of the 10-week intervention, teachers 
would continue to implement the model for the duration of 
the academic year (and into future years). Teachers partici- 
pated in sessions ten 2-hour sessions during the same 
10-week period that children participated in ten 45-minute 
sessions with facilitators. 

Students in study schools received the intervention across 
two consecutive years while enrolled in kindergarten and 
first grade. Kindergarten and first-grade teachers received 
the intervention for 10 weeks during one academic year, 
when students in the target cohort were enrolled in their 
classroom. Parents of consented students had an opportunity 
to participate in 10 weeks of parent intervention as well dur- 
ing their child’s kindergarten or first grade year. 


Facilitator training. INSIGHTS was delivered by a set of 
facilitators trained and supervised by the program developer. 
Prior to conducting the intervention in schools, all facilita- 
tors attended a graduate-level course to learn the theory and 
research underlying the JVS/GHTS intervention. New facilita- 
tors were mentored by experienced facilitators and supervised 
by the developer. Each facilitator conducted the comprehen- 
sive intervention in the school to which she or he was assigned. 


Program delivery: Focus on teachers. Teachers attended 
10 weekly 2-hour sessions based on a structured curriculum 
that included didactic content and professionally produced 
vignettes as well as handouts and group activities. Teachers 
were given assignments based on program content to com- 
plete between sessions. When necessary, make-up sessions 
were offered. Teachers received professional development 
credit and $40 gift cards for each session attended. During 
the same 10 weeks, the classroom program was delivered in 
45-minute lessons to all students enrolled in the classrooms 
of participating kindergarten and first-grade teachers. Teach- 
ers practiced lessons with any students who missed a session. 


Fidelity. To maintain model fidelity, facilitators followed 
scripts, used material checklists, documented sessions, and 
received ongoing training and supervision. Deviations or 
clinical concerns were discussed weekly in meetings with 
the program developer (see E. E. O’Connor et al., 2014). 
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Supervision was focused on challenges related to conduct- 
ing sessions, implementation logistics, and participant con- 
cerns. Teacher sessions were also videotaped and reviewed 
for coverage of content and effectiveness of facilitation 
(Hulleman & Cordray, 2009). Fidelity coding revealed that 
94% of the curriculum was adequately covered. 


INSIGHTS dosage. Dosage was assessed using program 
facilitators’ reports of number of sessions attended by teachers, 
children, and parents. Because we are leveraging data from all 
students who were enrolled in study schools, we do not have 
complete student- and parent-level dosage data for our full 
sample. However, we do have data for the subsample of stu- 
dents in these schools who were selected to participate in data 
collection activities and use that information as a way to esti- 
mate dosage across the broader group of student participants. 

As noted above, kindergarten teachers were invited to 
participate in 10 INSIGHTS sessions during the year that the 
target cohort of children in the school were enrolled in kin- 
dergarten; first-grade teachers were invited to participate in 
10 sessions in the year when that target cohort was in first 
grade. Children were invited to participate during both years. 
Dosage for teachers and children was high across schools. 
The average number of teacher sessions attended was 9.44 
(SD = 0.91) across kindergarten and first grade. Most teach- 
ers attended all 10 sessions (70.6%) and another 26.5% 
attended eight or nine sessions. There were no teacher-level 
differences in participation by teacher grade. Children 
attended 8.30 sessions on average (SD = 2.25) in each grade. 
Thirty-two percentage of these children were present for all 
classroom sessions, and 46.3% were present for 8 or 9 ses- 
sions across both kindergarten and first grade. 

Parents were invited to participate in 10 JNSIGHTS ses- 
sions when their child was enrolled in either kindergarten or 
first grade. Parent participation was lower than participation 
in the teacher and child components of the program. Although 
both program facilitators and program staff (recruiters and 
field managers) made all possible efforts to engage parents in 
INSIGHTS sessions, the average number of sessions eligible 
parents attended across years was 5.93 (SD = 4.15). Twenty- 
five percent of the parents were present for all sessions and 
38% were present for 8 sessions or more. This means that 
only about 10% of treatment group students in the current 
study—leveraging the larger data set—had parents partici- 
pate in at least 80% of parent sessions. Follow-up work done 
by the team using causal modeling to understand the added 
value of the parent program showed that the effects of the 
program were similar regardless of whether the parents par- 
ticipated or not (see McCormick, Cappella, O’Connor, & 
McClowry, 2016). This study showed that the children who 
were more disadvantaged—and perhaps stood to gain the 
most from the program—were most likely to have parents 
who did not participate. These children still benefited sub- 
stantially from the teacher and child program sessions even in 
the absence of the parent component. 
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Attention-Control Condition 


A supplemental reading program served as the attention- 
control condition. The rationale for having an attention-control 
condition, rather than a business as usual condition, was to test 
whether it was truly the SEL programming provided by 
INSIGHTS that changed student outcomes, or simply having 
more attention and opportunities for professional development. 
In addition, the program provided the schools in low-income 
neighborhoods with additional literacy-related resources and 
allowed for a conservative estimate of intervention effects. 
There was no overlap in content covered in the supplemental 
reading program and the JVS/GHTS intervention. 

Teachers in attention-control schools attended two sepa- 
rate workshops, each 2 hours long, in which strategies to 
enhance literacy were presented and reading materials for 
the children were provided. Teachers received professional 
development credit and $40 for classroom resources for each 
workshop. Reading program facilitators conducted the 
weekly reading sessions directly with students across the 
10-week program period. Fidelity checklists revealed that 
95% of topics were covered. 


Appendix B 
Missing Data Analysis 


The total analytic sample size for this study is N= 1,634 
students. Of the total number of students, 1,263 (77%) 
remained in the sample through the end of the fifth grade, 
while 371 (23%) are missing outcome data because they 
attrited from the school district before the end of the fifth 
grade. Chi-square tests indicated that those who left the 
school district by fifth grade were less likely to be eligible 
for free or reduced price lunch in both the treatment (77 = 
14.63, p < .01) and attention-control (y? = 16.14, p < .01) 
groups. Children who left the district were also less likely 
to have attended a public preschool program in both the 
treatment (y* = 33.28, p < .01) and attention-control 
groups (y° = 39.43, p < .01). Children enrolled in 
INSIGHTS schools attrited at the same rate as children 
enrolled in attention-control schools. Within the nonat- 
trited sample, there were N = 639 children in the attention- 
control group (76% of the baseline attention-control 
group) and NV = 624 children in the INSIGHTS group (78% 
of the baseline treatment group). We used a cross-tab anal- 
ysis with a chi-square test (y° = 32.87, p = 56) to deter- 
mine that the rate of attrition was not statistically different 
across these two groups. According to the standards from 
the What Works Clearinghouse (2017), this study consti- 
tutes a low-attrition RCT, meaning that overall attrition 
and differential attrition indicate a tolerable level of bias 
for the estimated intervention effect. Given these moder- 
ately low levels of attrition and minimal differential attri- 
tion between the treatment and control groups, we decided 


not to impute outcome data. The students in the treatment 
group are similar to the control group students that did not 
attrit and are included in the impact analysis. See Table 1 
in the main text for further details on the nonattrited 
sample. 

Covariate data had minimal levels of missingness given 
that these variables were collected when parents enrolled 
their children in public school. However, we did compare the 
students missing covariate data from those who had complete 
covariate data. We found that 14.45% of the sample with 
nonmissing outcome data was missing one or more covari- 
ates. Students missing covariate data were more likely to be 
White, to not have attended PreK, to be in the second cohort 
of the study, and to be in the control group. Using the assump- 
tion that data were Missing at Random, the team used a mul- 
tiple data imputation method in STATA MICE and imputed 
100 data sets in order to generate complete data on all covari- 
ates (Graham et al., 2007). Final parameter estimates were 
generated by calculating the mean across estimates from all 
100 data sets. All predictive analyses presented in this article 
use the imputed data. A set of robustness checks demon- 
strated that the results from the multiple imputation proce- 
dure mirror the results using a listwise deletion approach with 
a smaller sample. 


Appendix C 
Model Specifications 


The first impact model we examined in this study (Model 
2 in Tables 3 and 4 in the main text) is as follows. This model 
includes only an intercept (8) and a Level-2 indicator vari- 
able for assignment to the INSIGHTS treatment (8 a): The 
model includes a random intercept for study school ) and 
a residual error term (é,). 


logit {Pr(, = 1x4» Gi} = B, + B,JNSIGHTS,+ C,+ «, 


Next, we added a series of Level-1 covariates to this model 
to improve the precision of the impact estimates. Results are 
illustrated in the panel labeled Model 3 in Tables 3 and 4. 


logit {Pr(yj = 1x3. G;)} = Bi + ByNSIGHTS, + 
6, BASELINECHILDAGE, + 
B,FEMALE, +B;BLACK, + 
B, HISPANIC, +B,LOW — SES, + 
B, DLL, +B, IMMIGRANT, + 
B,,PREKATTEND, +C,+ &, 


Finally, we added a series of Level-2 covariates to this model 
to further improve the precision of the impact estimates and 
adjust for any small group-based differences in school-level 
demographic characteristics. Results are illustrated in the 
panel labeled Model 4 in Tables 3 and 4. 
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logit {Pr( ¥,= lle i} = B, + B,JNSIGHTS, + 
B,BASELINECHILDAGE, + 
B, FEMALE, +B,BLACK, + 
B, HISPANIC, +B,LOW — SES, + 
B,DLL, + B,[MMIGRANT, + 
B,)PREKATTEND, + 
B,,COHORT2, + 
B,,COHORT3, + 
B,,SCHOOLSIZE, + 
B,,%ELACOMPETENT, + 
B,;%MATHCOMPETENT, + 
B,.ATTENDANCE, +6, + &, 


Appendix D 
Summary of Additional Robustness Checks 


Findings from models using logistic regressions with 
clustered standard errors for study school are included in 
Appendix D Tables D1 and D2. As illustrated, impact find- 
ings appeared fairly similar using this model specification. 
However, the standard errors on the treatment coefficients 
and treatment x low-income interaction terms are a bit larger 
in these models, relative to the random intercepts model 
specification. In addition, the main effect of INSIGHTS on 
receipt of SPED services was only marginally significant at 
p = .08, when we used this specification (OR = 0.82, 
log(odds) = —0.20, SE = 0.11, CI [-0.42, 0.02], p = .08). The 
moderated effect for low-income students was robust to this 
specification at the p < .05 level (OR = 0.33, log(odds) = 
—1.11, SE = 0.43, CI [-1.95, —0.26], p = .03). 

Furthermore, given that we were missing 23% of outcome 
data, we conducted a bounding exercise in line with recommen- 
dations by Lee (2009) using the Bounds package in STATA to 
estimate worst-case bounds on regressions with some observa- 
tions missing outcome and covariate data. This estimation cal- 
culates a confidence interval around an upper and lower bound 
of the impact estimate. Using this strategy, we found that the 
log(odds) of the lower bound of the impact estimate on receipt 
of SPED services was —0.11 (OR = 0.89, SE(log(odds)) = 0.06, 
CI [-0.23, 0.01]). The log(odds) odds of the upper bound of the 
impact estimate was —.20 (OR = 0.82, SE [log(odds)] = 0.09, CI 
[—0.38, —0.02]). As such, the lower bound was no longer statis- 
tically significant for the main effect of JIVS/GHTS on receipt of 
SPED services. Effects for the moderation findings were more 
consistent with a lower bound of —0.62 (OR = 0.54, 
SE(log(odds)) = 0.28, CI [-1.18, —0.06]) and an upper bound of 
—1.31 (OR = 0.27, SE(log(odds)) = 0.44, CI [-2.19, 0.43]), 
which were both statistically significant at the p < .05 level. 

Next, we tested whether our results changed when we 
group-mean centered the Level-1 covariates in our analyses. 
Enders and Tofighi (2007) argue that centering Level-1 
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variables can improve statistical inference when one is using 
a two-level random effects model where the variable of 
interest—assignment to INSIGHTS in our case—is opera- 
tionalized at Level 2. We found that our impact results for 
both grade retention and receipt of SPED services replicated 
when using this approach (see Appendix D Tables D3 and 
D4, respectively). The point estimates for the impacts were 
consistent and the standard errors and corresponding p val- 
ues were actually a bit smaller in these models than in our 
original mixed-effects logistic regressions where we did not 
group-mean center the Level-1 covariates. We retain the 
uncentered versions of the Level-1 covariates in the presen- 
tation of our main set of results in order to maintain simple 
interpretation of effects. 


In addition, we tested to see whether the results from our 
mixed-effects regression were sensitive to the choice of the 
covariance structure. As noted above, we found that our 
results were consistent regardless of whether we tested the 
mixed-effects regression using an unstructured, indepen- 
dent, exchangeable, or identity structure. Finally, we exam- 
ined whether the results were consistent when we added 
school-level measures of the individual-level demographic 
covariates. We tested a final set of models where we further 
included covariates at the school-level for percent female, 
percent Black, percent Hispanic, percent low SES, percent 
dual language learner, % immigrant, and % students who 
attended PreK. Impact findings were all robust to the further 
inclusion of these school-level demographic characteristics. 


TABLE D1 
Impacts of INSIGHTS on Grade Retention (Logistic Regression With Clustered Standard Errors) 
Model 1 Model 2 Model 3 Model 4 
Robust Robust Robust Robust 

Fixed effects OR  Log(odds) SE OR  Log(odds) SE OR  Log(odds) SE OR  Log(odds) SE 
Intercept 0.28 -1.29 0.13 0.32 =1,15 0.19 0.22 =1,52 0.41 0.22 =1,52 0.41 
Treatment (INSIGHTS) 0.77 —0.26 0.27 0:79. =0:23 0.26 0.79 —0.24 0.24 
Baseline age 0.51 —0.68** 0.16 1.97 0.68** = 0.12 
Female 0.84 -0.18 0.15 0.84 —0.18 0.16 
Black 1.07 0.07 0.22 1.07 0.07 0.27 
Hispanic 1.08 0.08 0.20 1.08 0.08 0.31 
Low SES 2.41 0.88** 0.17 2.41 0.88** 0.18 
Dual language learner 0.63 —0.46* 0.22 0.63 —0.46 0.29 
From recent immigrant family 0.91  —0.09 0.29 0.91 —0.09 0.28 
Prekindergarten attender 0.36 —1.01** 0.17 0.36 —101** O15 
Cohort 2 1.43 0.36* 0.14 
Cohort 3 LD 0.17 0.21 
School size 1.21 0.19 0.27 
School % competent ELA test 0.84 =0.17* 0.09 
School % competent math test 0.82 —0.20* 0.09 
School average attendance 0.94 —0.06 0.11 
Fit statistics 

AIC 362.17 361.54 349.64 342.54 

BIC 372.53 371.84 338.53 329.48 
Log likelihood —184.35 —170.42 —169.04 —159.47 
Note. OR = odds ratio; SE = standard error; SES = socioeconomic status; AIC = Akaike information criterion; BIC = Bayesian information criterion. 
“p< .10. *p <.05. **p <.01. 
TABLE D2 
Impacts of INSIGHTS on Receipt of SPED Services (Logistic Regression With Clustered Standard Errors) 

Model 1 Model 2 Model 3 Model 4 
Robust Robust Robust Robust 

Fixed effects OR  Log(odds) SE OR Log(odds) SE OR  Log(odds) SE OR  Log(odds) SE 
Intercept 0:10 -2:29%* 0.19 0.09 —2,38** 0.08 0.11 =2,19** 0.08 0.11 =2.18** 0.08 
Treatment (INSIGHTS) 0.83 -0.19" 0.12 0.81 -0.21* 0.12 0.82 -0.20° 0.11 
Baseline age 1.21 0.19 0.13 1.20 0.18 0.13 
Female 0.29 1.25% 0.22 0.29 —1.25** 0.25 
Black 3.46 1.24** 0.33 3.53 1.26** 0.36 
Hispanic 3.60 1.28** 0.47 = 3.39 1.22* 0.52 
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(continued) 


TABLE D2 (CONTINUED) 


Model 1 Model 2 Model 3 Model 4 
Robust Robust Robust 

Fixed effects OR  Log(odds) Log(odds) SE OR  Log(odds) SE OR  Log(odds) SE 
Low SES 1.75 0.56** 0.21 1.72 0.54* 0.25 
Dual language learner 0.54 —0.61* 0.24 0.53 —0.63* 0.28 
From recent immigrant 0.16 —1.86* 0.91 0.16 —1.82* 0.89 
family 
Prekindergarten attender 0.90 =0:11 0.14 087 -0.14 0.13 
Cohort 2 0.78  —0.25 0.25 
Cohort 3 — 0.84 -0.17 0.31 
School size — 1.25 0.22* 0.09 
School % competent ELA = 1.08 0.08 0.25 
test 
School % competent math —_— 0.90 -0.11 0.23 
test 
School average attendance — 1.28 0.25 0.21 
Fit statistics 

AIC 218.43 217.54 211.57 206.54 

BIC 226.43 210.54 204.65 176.44 

Log likelihood —107.54 —101.04 —97.74 —96.44 


Note. OR = odds ratio; SE = standard error; SES = socioeconomic status; AIC = Akaike information criterion; BIC = Bayesian information criterion; SPED = special education. 


*p< 10. *p<.05. **p < 01. 


TABLE D3 


Impacts of INSIGHTS on Grade Retention (Group-Mean Centered L1 Covariates) 


Fixed effects OR Log(odds) SE 
Intercept 0.24 —1.43** 0.22 
Treatment (INSIGHTS) 0.77 —0.26 0.18 
Student-level covariates 
Baseline age (centered) 0.61 —0.49** 0.09 
Female (centered) 0.74 —0.30** 0.12 
Black (centered) 1,12 0.11 0.35 
Hispanic (centered) 1.38 0.32 0.49 
Low SES (centered) 2.01 0.70** 0.31 
Dual language learner (centered) 0.62 —0.48* 0.17 
From recent immigrant family (centered) 0.74 —0.30 0.29 
Prekindergarten attender (centered) 0.52 —0.65** 0.09 
School-level covariates 
Cohort 2 1.22 0.20 0.25 
Cohort 3 1.19 0.17 0.23 
School size 1.36 0.31 0.28 
School % competent ELA test 0.88 —0.13* 0.06 
School % competent math test 0.82 —0.20** 0.05 
School average attendance 0.86 —-0.15 0.15 
Variance components Variance SE 
Baseline study school 0.05 0.04 
Fit statistics 
AIC 319.57 
BIC 335.37 
Log likelihood —159.47 


Note. OR = odds ratio; SE = standard error; SES = socioeconomic status; AIC = Akaike information criterion; 


*p<.10. *p <.05. **p < 01. 


BIC = Bayesian information criterion. 
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TABLE D4 


Impacts of INSIGHTS on Receipt of SPED Services (Group-Mean Centered L1 Covariates) 


Fixed effects OR Log(odds) SE 
Intercept 0.11 =2.21%* 0.09 
Treatment (INSIGHTS) 0.83 =0:19* 0.06 
Student-level covariates 
Baseline age (centered) 1.09 0.09 0.25 
Female (centered) 0.46 —0.78** 0.09 
Black (centered) 0.14 —1.97* 0.91 
Hispanic (centered) 0.33 —1.11 0.82 
Low SES (centered) 0.27 =1,31* 0.56 
Dual language learner (centered) 1.91 0.65 0.46 
From recent immigrant family (centered) 0.61 —0.49 0.28 
Prekindergarten attender (centered) 0.75 —0.29 0.18 
School-level covariates 
Cohort 2 1.13 0.12 0.42 
Cohort 3 0.12 =212** 0.55 
School size 1.33 0.29* 0.12 
School % competent ELA test 0.96 —0.04 0.22 
School % competent math test 0.92 —0.08 0.32 
School average attendance 1.09 0.09 0.27 
Variance components Variance SE 
Study school 0.04* 0.02 
Fit statistics 
AIC 166.75 
BIC 182.57 
Log likelihood —94.54 


Note. OR = odds ratio; SE = standard error; SES = socioeconomic status; AIC = Akaike information criterion; BIC = Bayesian information criterion. 


*p<.10. *p<.05. **p< 01. 
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Notes 


1. We only include children enrolled in study schools at the time of 
random assignment in this analysis to maintain the internal validity of 
the design. These students were exposed to the intervention or compar- 
ison condition in kindergarten and first grade, assuming they remained 
in the school for both consecutive years. However, there were students 
who are not included in the study sample who did receive the interven- 
tion in first grade. We do not include those students in analyses because 
they enrolled in schools after random assignment. 

2. Earlier work from the INSIGHTS randomized control trial 
aimed to use a consented sample to collect data on students and 
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teachers and estimate the short-term impacts of the program on 
a host of outcomes at multiple levels (see E. E. O’Connor et al., 
2014). That work focused on a subset of students participating 
in the intervention whose parent consented for them to take part 
in data collection activities. We limited the number of students 
enrolled in the study in each classroom (4-10 students per class- 
room was the average) in order to reduce burden on teachers for 
direct assessments and teacher reports. The consented sample for 
the study does not differ from the sample enrolled in the current 
study, which uses de-identified and thus more complete data. 

3. We did aim to enroll schools in the study who had at least 
75% of students eligible for free or reduced-price lunch. However, 
at the time that this study was being conducted in Brooklyn, NY, 
we observed a massive wave of gentrification where higher income 
families were moving into traditionally lower income neigh- 
borhoods and enrolling their children in neighborhood schools. 
Accordingly, although we originally sought to conduct the study 
only within lower income schools, the student sample has sufficient 
variation in order to consider subgroup effects for children who 
were and were not eligible for free or reduced-price lunch. Many 
of the schools that the study was originally conducted in did transi- 
tion from being made up of primarily free lunch eligible students in 
2008-2010 to primarily non-free lunch eligible by 2018. Although 


this gentrification process has raised a number of challenges 
more generally for neighborhoods and schools in Brooklyn, it has 
allowed us to consider variation in effects across a more diverse 
group than we otherwise would have been able to do. 
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